Skip to main content

All-in-One Log Collection Platform New Relic Experience (Cloudflare Workers Log Push / Java Log Push)

1. Introduction​

While researching what interesting things Cloudflare has to offer, I suddenly wondered if Cloudflare Workers logs could be aggregated together. Then I discovered that Cloudflare has a feature to push logs to other platforms. After trying them one by one, I found that New Relic is absolutely amazing - it can collect logs from various applications across different platforms all in one place.

2. New Relic Overview​

A quick Google search describes it as: New Relic is a full-stack observability platform that provides a single source of truth for application performance, infrastructure monitoring, log management, error tracking, real user monitoring, and more.

2.1 Aggregate and Manage Logs from Various Platforms and Applications​

The integrations are so numerous they can't fit on one page:

There are many categories:

2.2 The Key Point: New Users Get 100GB of Log Storage​

Unlike some cloud storage services that stingily give you only 5GB, new users get 100GB:

Let me translate all the benefits:

  • One full platform user

    • Access to all 30+ platform features, including APM with tracing and logs.

  • 100 GB data ingest

    • Includes security data analyzed by New Relic Vulnerability Management.

  • Default log obfuscation

    • For masking known credit card and social security number patterns in logs.

  • Unlimited basic users

    • Access to data ingest, queries, dashboards, and alerts.

  • Default data retention

    • At least eight days to recreate issues and better understand performance.

  • Unlimited Ping monitors

    • Includes 500 synthetic monitoring checks deployed to meet your needs.

With all that said, I tried pushing logs from the following platforms. Before proceeding with the operations below, you'll naturally need to register a New Relic account.

3. Push CF Worker Logs to New Relic​

tip

Before proceeding, you should know that Cloudflare's Logpush to other platforms is included in the Workers Paid plan. The Free plan may not have this feature.

See this page for details: Logpush

These are the limits seen under the Workers Paid plan:

3.1 Configuration​

In the New Relic dashboard, click Integrations & Agents, search for Cloudflare to find the corresponding integration.

The integration method is straightforward. Select Cloudflare Dashboard, then after creating a license key in the next step, go back to the CF interface and find the option to push logs to New Relic.

Fill in the endpoint required by New Relic and the license key you just created.

The logs to push are naturally CF Workers logs.

I selected all logs here.

After completion, you can see that log push is now active.

Return to New Relic, click Test Connection, and it should succeed.

3.2 Enable Your CF Workers Push​

However, at this point, no actual logs are being pushed to New Relic yet. We still need to configure each worker.

The items in the red box need to be enabled:

3.3 Check Log Push​

After the above preparations are ready, trigger your workers. If you can already see logs in the workers log interface, then logs should also be pushed to New Relic.

Return to New Relic, click Logs in the left sidebar, then select All logs, and you'll see your CF Workers logs!

4. Push Docker + Java Application Logs to New Relic​

Pushing Java logs is similar: Integrations & Agents -> Java -> Docker

4.1 Enter Your License Key, Then Go to the Download your Java agent files Step​

Note that pushing Java logs requires downloading their Java agent and including it in your project. That's the zip file downloaded with curl in the image. Extract it and place it in your project's root directory.

Then download the newrelic.yml containing the license key and overwrite the file with the same name in the extracted folder.

For example:

4.2 Almost Done - Modify Your Dockerfile When Starting Your Java Application​

Below is my Java project's Dockerfile for your reference. The highlighted parts are the changes:

FROM azul/zulu-openjdk:17.0.13

ARG JAR_NAME
ENV PROJECT_NAME ${JAR_NAME}
ENV PROJECT_HOME /usr/local/${PROJECT_NAME}

RUN mkdir -p /usr/local/newrelic
ADD ./newrelic/newrelic.jar /usr/local/newrelic/newrelic.jar
ADD ./newrelic/newrelic.yml /usr/local/newrelic/newrelic.yml


RUN ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime
RUN echo 'Asia/Shanghai' >/etc/timezone
RUN mkdir $PROJECT_HOME && mkdir $PROJECT_HOME/logs

ARG JAR_FILE
COPY ${JAR_FILE} $PROJECT_HOME/${JAR_NAME}.jar


ENTRYPOINT java -javaagent:/usr/local/newrelic/newrelic.jar -jar -XX:+UseZGC $PROJECT_HOME/$PROJECT_NAME.jar

The optional step can be skipped directly and won't affect log pushing. Description of the optional step:

You can run a containerized version of the infrastructure monitoring agent to monitor container metrics and the underlying host. Use the agent configuration containing your New Relic license key. More configuration options for the New Relic infrastructure agent are detailed here.

4.3 Rebuild the Image​

The final step is naturally to rebuild the image according to your Dockerfile and start it. After starting, pay attention to whether your container is running normally and whether logs are normal.

Return to New Relic's Logs interface, and you'll see your logs.